45 research outputs found
Neural Networks for Predicting Algorithm Runtime Distributions
Many state-of-the-art algorithms for solving hard combinatorial problems in
artificial intelligence (AI) include elements of stochasticity that lead to
high variations in runtime, even for a fixed problem instance. Knowledge about
the resulting runtime distributions (RTDs) of algorithms on given problem
instances can be exploited in various meta-algorithmic procedures, such as
algorithm selection, portfolios, and randomized restarts. Previous work has
shown that machine learning can be used to individually predict mean, median
and variance of RTDs. To establish a new state-of-the-art in predicting RTDs,
we demonstrate that the parameters of an RTD should be learned jointly and that
neural networks can do this well by directly optimizing the likelihood of an
RTD given runtime observations. In an empirical study involving five algorithms
for SAT solving and AI planning, we show that neural networks predict the true
RTDs of unseen instances better than previous methods, and can even do so when
only few runtime observations are available per training instance
Hyperparameters in Reinforcement Learning and How To Tune Them
In order to improve reproducibility, deep reinforcement learning (RL) has
been adopting better scientific practices such as standardized evaluation
metrics and reporting. However, the process of hyperparameter optimization
still varies widely across papers, which makes it challenging to compare RL
algorithms fairly. In this paper, we show that hyperparameter choices in RL can
significantly affect the agent's final performance and sample efficiency, and
that the hyperparameter landscape can strongly depend on the tuning seed which
may lead to overfitting. We therefore propose adopting established best
practices from AutoML, such as the separation of tuning and testing seeds, as
well as principled hyperparameter optimization (HPO) across a broad search
space. We support this by comparing multiple state-of-the-art HPO tools on a
range of RL algorithms and environments to their hand-tuned counterparts,
demonstrating that HPO approaches often have higher performance and lower
compute overhead. As a result of our findings, we recommend a set of best
practices for the RL community, which should result in stronger empirical
results with fewer computational costs, better reproducibility, and thus faster
progress. In order to encourage the adoption of these practices, we provide
plug-and-play implementations of the tuning algorithms used in this paper at
https://github.com/facebookresearch/how-to-autorl
Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates
The optimization of algorithm (hyper-)parameters is crucial for achieving
peak performance across a wide range of domains, ranging from deep neural
networks to solvers for hard combinatorial problems. The resulting algorithm
configuration (AC) problem has attracted much attention from the machine
learning community. However, the proper evaluation of new AC procedures is
hindered by two key hurdles. First, AC benchmarks are hard to set up. Second
and even more significantly, they are computationally expensive: a single run
of an AC procedure involves many costly runs of the target algorithm whose
performance is to be optimized in a given AC benchmark scenario. One common
workaround is to optimize cheap-to-evaluate artificial benchmark functions
(e.g., Branin) instead of actual algorithms; however, these have different
properties than realistic AC problems. Here, we propose an alternative
benchmarking approach that is similarly cheap to evaluate but much closer to
the original AC problem: replacing expensive benchmarks by surrogate benchmarks
constructed from AC benchmarks. These surrogate benchmarks approximate the
response surface corresponding to true target algorithm performance using a
regression model, and the original and surrogate benchmark share the same
(hyper-)parameter space. In our experiments, we construct and evaluate
surrogate benchmarks for hyperparameter optimization as well as for AC problems
that involve performance optimization of solvers for hard combinatorial
problems, drawing training data from the runs of existing AC procedures. We
show that our surrogate benchmarks capture overall important characteristics of
the AC scenarios, such as high- and low-performing regions, from which they
were derived, while being much easier to use and orders of magnitude cheaper to
evaluate
AutoML in Heavily Constrained Applications
Optimizing a machine learning pipeline for a task at hand requires careful
configuration of various hyperparameters, typically supported by an AutoML
system that optimizes the hyperparameters for the given training dataset. Yet,
depending on the AutoML system's own second-order meta-configuration, the
performance of the AutoML process can vary significantly. Current AutoML
systems cannot automatically adapt their own configuration to a specific use
case. Further, they cannot compile user-defined application constraints on the
effectiveness and efficiency of the pipeline and its generation. In this paper,
we propose Caml, which uses meta-learning to automatically adapt its own AutoML
parameters, such as the search strategy, the validation strategy, and the
search space, for a task at hand. The dynamic AutoML strategy of Caml takes
user-defined constraints into account and obtains constraint-satisfying
pipelines with high predictive performance
The Configurable SAT Solver Challenge (CSSC)
It is well known that different solution strategies work well for different
types of instances of hard combinatorial problems. As a consequence, most
solvers for the propositional satisfiability problem (SAT) expose parameters
that allow them to be customized to a particular family of instances. In the
international SAT competition series, these parameters are ignored: solvers are
run using a single default parameter setting (supplied by the authors) for all
benchmark instances in a given track. While this competition format rewards
solvers with robust default settings, it does not reflect the situation faced
by a practitioner who only cares about performance on one particular
application and can invest some time into tuning solver parameters for this
application. The new Configurable SAT Solver Competition (CSSC) compares
solvers in this latter setting, scoring each solver by the performance it
achieved after a fully automated configuration step. This article describes the
CSSC in more detail, and reports the results obtained in its two instantiations
so far, CSSC 2013 and 2014